The impact of image descriptions on user tagging behavior: A study of the nature and functionality of crowdsourced tags

Crowdsourcing has been emerging to harvest social wisdom from thousands of volunteers to perform series of tasks online. However, little research has been devoted to exploring the impact of various factors such as the content of a resource or crowdsourcing interface design to user tagging behavior. While images’ titles and descriptions are frequently available in image digital libraries, it is not clear whether they should be displayed to crowdworkers engaged in tagging. This paper focuses on offering an insight to the curators of digital image libraries who face this dilemma by examining (i) how descriptions influence the user in his/her tagging behavior and (ii) how this relates to the (a) nature of the tags, (b) the emergent folksonomy, and (c) the findability of the images in the tagging system. We compared two different methods for collecting image tags from Amazon’s Mechanical Turk’s crowdworkers – with and without image descriptions. Several properties of generated tags were examined from different perspectives: diversity, specificity, reusability, quality, similarity, descriptiveness, etc. In addition, the study was carried out to examine the impact of image description on supporting users’ information seeking with a tag cloud interface. The results showed that the properties of tags are affected by the crowdsourcing approach. Tags from the “with description” condition are more diverse and more specific than tags from the “without description” condition, while the latter has a higher tag re-use rate. A user study also revealed that different tag sets provided different support for search. Tags produced “with description” shortened the path to the target results, while tags produced without description increased user success in the search task. Keywords: Crowdsourcing, image description, tagging behavior, Amazon Mechanical Turk, image search. The Impact of Image Descriptions on User Tagging Behavior: A Study of the Nature and Functionality of Crowdsourced Tags 1. Introduction Crowdsourcing has emerged as a popular modern approach to perform information processing tasks that are difficult or impossible to automate. Among other applications, crowdsourcing has become a powerful mechanism to harvest collective wisdom from thousands of volunteers (Howe, 2008; Surowiecki, 2004). A good example of this kind of task is image annotation with keywords (known as image tagging). Keywords are critical for finding images, yet computers cannot tag images automatically. The need to annotate images in digital libraries and other image collections has encouraged researchers and practitioners to explore a range of crowdsourcing approaches to collect image tags. This idea was pioneered by early image sharing systems (e.g., Flickr) and was later scaled up using game-based approaches (e.g., ESP game) and paid crowdsourcing marketplaces (e.g, Amazon’s Mechanical Turk) (Nowak & Rüger, 2010; Sorokin & Forsyth, 2008). The importance of tag crowdsourcing, in turn, encouraged a stream of research focused on the quality and other properties of crowdsourced tags. This research was stimulated by the early work of Golder & Huberman (2006) and Kowatsch & Maass (2008), who discovered that the quality and diversity of crowdsourced tags can be affected by various components of the tagging interface, such as presentation of current and recommended tags. Following this discovery, a few other teams explored user-tagging 1 ESP game collects image metadata by engaging users in an image tagging game which was originally conceived by Luis von Ahn of Carnegie Mellon University. Google bought a license in 2006 to increase the keywords of images for its online image search. 2 Amazon Mechanical Turk is a crowdsourcing Internet marketplace where individuals or businesses can recruit human intelligence to perform tasks that computer are not able to do. behavior and the impact of various parameters on the properties of produced tags. The study presented in this paper extends this research by examining the impact of image description on the nature and the quality of crowdsourced tags. This study is important from both theoretical and practical perspectives. Image tagging context with non-textual primary content and secondary textual descriptions is considerably different from the already-explored context characterized by primary textual content and tag recommendations. Existing empirical data and models are not sufficient to reliably predict the impact of image descriptions on tag production. New empirical data has to be collected to expand and generalize known models. On the practical side, the study is important to guide managers of image collections in the process of tag crowdsourcing. While image descriptions are frequently available in image collections, it is not clear whether or not they should be displayed to crowdworkers engaged in tagging. On the one hand, the presence of image description could help the crowdworkers to generate more tags and to make them more specific. On the other hand, it could curb their creativity and harm the diversity of the resulting tags. The goal of our study was to collect and analyze empirical data on the impact of image descriptions on tagging to advance both our understanding of the tagging process and the current practice of tag crowdsourcing in image collections. Due to the lack of relevant models that describe the process, we designed the study in an open format. That is, instead of attempting to prove or disprove outcomes predicted by the existing theory, we adopted an empirical approach that is common for examining human-computer interfaces and formulated our research questions in the following way: 3 The quality of crowdsourced tags in this context is considered from the perspective of tag usage, which contains tag reusability (Nowak & Rüger, 2010; Sen et al., 2006), resource findability and resource discrimination (Dellschaft & Staab, 2012). • RQ1: How descriptions influence the user in his tagging? • RQ2: How this relates to the (a) nature of the tags, (b) the emergent folksonomy, and (c) the findability of the images in the tagging system? According to the study of Nowak and Ruger (2010), tags crowdsourced via Amazon’s Mechanical Turk are less costly, yet as reliable as expert-level tags. Therefore, this study compared two different methods for collecting image tags from Amazon’s Mechanical Turk crowdworkers – with and without image descriptions. We investigated the properties of generated tags from different perspectives including generality, quality, similarity, and descriptiveness. We also conducted a user study on Amazon’s Mechanical Turk to compare users’ image search performance using tags sets produced with and without descriptions. The remainder of the paper starts with the analysis of similar work (Section 2) and follows by presenting three components of our study. Section 3 explains the process of tag crowdsourcing for our study. It introduces the image collection, presents two interfaces for tag crowdsourcing (with and without descriptions) and provides some information about produced datasets. Section 4 focuses on the qualitative comparison of image tags crowdsourced with and without descriptions. It examines density, generality, consistency, and other properties of tag collections. Section 5 examines the practical differences between two collected tag sets by comparing their impact on image findability in a user study. Finally, Section 6 completes the paper with a discussion of the results and limitations of the present study.